跳转至

文章背景与核心概要

带可验证奖励的强化学习（RLVR）显著提升了多模态大模型（MLLMs）的准确性，但这些性能收益通常非常脆弱。诸如改写问题或更改提示词模板等简单的输入修改，经常会导致模型性能下降，这给医疗视觉问答（VQA）等高风险领域的部署带来了可靠性风险。

为了解决这一问题，作者将这种脆弱性归因于标准强化学习目标中的两个主要缺陷：1. 格式与内容的混淆；2. 分布狭窄。为了构建更具韧性的后训练范式，作者提出了“提示词不变强化学习”（Prompt-Invariant RLVR, PIRL）。PIRL 集成了动态三元奖励（以区分格式与语义）和一个一致性正则化器（利用嵌入空间对抗性攻击来强制模型在语义等价的提示词之间保持策略不变性）。

在严格的压力测试下，PIRL 将平均基准测试准确率的下降幅度限制在 \(\le 1\%\)（相比之下，标准 GRPO 的降幅约为 \(3\%\)），并在动态评估中展现出卓越的抗干扰能力。

Improving Generalization Robustness of Multimodal RLVR

arXiv ID: 2608.08802
Primary Subject: Artificial Intelligence (cs.AI)
Publication Date: August 9, 2026 (Last revised August 14, 2026)

📌 Summary

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves the accuracy of Multimodal Large Language Models (MLLMs), but these performance gains are fragile. Simple input modifications—such as paraphrasing questions or altering prompt templates—frequently degrade model performance, posing reliability risks for deployment in high-stakes domains like medical Visual Question Answering (VQA).

To address this, the authors trace the vulnerability to two primary flaws in the standard RL objective: 1. Format vs. Content Conflation: Standard binary verifiers cannot distinguish between an incorrect answer and a correctly answered but improperly formatted response. 2. Distributional Narrowness: Training distributions cover only a small subset of real-world prompts, causing policies to falter when facing unseen variations during deployment.

To build a more resilient post-training paradigm, the authors propose Prompt-Invariant RLVR (PIRL). PIRL integrates a dynamic trinary reward (to separate format from semantics) and a consistency regularizer (leveraging an embedding-space adversary to enforce policy invariance across semantically equivalent prompts).

Under rigorous stress testing, PIRL limits average benchmark accuracy drops to \(\le 1\%\) (compared to roughly \(3\%\) for standard GRPO) and achieves superior resilience during dynamic evaluations.

Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves the accuracy of Multimodal Large Language Models (MLLMs), but these performance gains are fragile. Simple input modifications—such as paraphrasing questions or altering prompt templates—frequently degrade model performance, posing reliability risks for deployment in high-stakes domains like medical Visual Question Answering (VQA).

To address this, the authors trace the vulnerability to two primary flaws in the standard RL objective: 1. Format vs. Content Conflation: Standard binary verifiers cannot distinguish between an incorrect answer and a correctly answered but improperly formatted response. 2. Distributional Narrowness: Training distributions cover only a small subset of real-world prompts, causing policies to falter when facing unseen variations during deployment.

To build a more resilient post-training paradigm, the authors propose Prompt-Invariant RLVR (PIRL). PIRL integrates a dynamic trinary reward (to separate format from semantics) and a consistency regularizer (leveraging an embedding-space adversary to enforce policy invariance across semantically equivalent prompts).

Under rigorous stress testing, PIRL limits average benchmark accuracy drops to \(\le 1\%\) (compared to roughly \(3\%\) for standard GRPO) and achieves superior resilience during dynamic evaluations.

👥 Authors

  • Pengfei Zhou
  • Zhiwei Tang
  • Xiaopeng Peng
  • Chenrui Zhou
  • Lama Moukheiber
  • Yixing Ma
  • Bin Xu
  • Jiajun Song
  • Zhenglin Wan
  • Wangbo Zhao
  • Jiasheng Tang
  • Bohan Zhuang
  • Fan Wang
  • Yang You
  • Pengfei Zhou
  • Zhiwei Tang
  • Xiaopeng Peng
  • Chenrui Zhou
  • Lama Moukheiber
  • Yixing Ma
  • Bin Xu
  • Jiajun Song
  • Zhenglin Wan
  • Wangbo Zhao
  • Jiasheng Tang
  • Bohan Zhuang
  • Fan Wang
  • Yang You

📋 Additional Metadata